Background:
While trying to play catchup one of our Nodes got stuck at block 255 (886 blocks to catchup, also block 255 requires block 256 to get the consensus messages to commit when catching up).  We were able to do a full recovery by copying over the data from another node eventually after a lot of trial and tribulations.

During this process we discovered an interesting anomaly, Block 256 which originally got successfully published on 1/14, was re-attempted on 1/16 (and then several times each day since then). Same thing happened with block 257. And since yesterday Block 16 which was originally published 1/07 seems to be re-attempted as well. 

I am pasting some logs below, but really would be grateful if anyone could help us with understanding and potentially avoiding this scenario. (The network continues to work fine, but nodes catching up when they encounter such a block seem to stall in the catchup process.0

Validator version: 1.1.5
PBFT version: 1.0.1

Also, as you will notice from below, while the pBFT spec indicates that the block number is used as a sequence number in PBFT, when the older blocks were re-attempted you can see the Block number and sequence numbers do not match. 

And, there is no non-determinism in our TP. We have tested over several months (and our other environments do not show the same behavior having processed a few thousand blocks so far).

Validator Log (for block 256).
[2020-01-14 04:24:15.145 INFO     block_validator] Block 10b2eac6d898f20a015922565a8c31bc739b4392a9fbda27b58b5d035f5516b3608e13757db2e291babfa51bed2dd5543fb5bd607bd2c1acf2cf09c23b995fa0 (block_num:255, state:6310d7f4821fe31b9453d44e4bd202c8c5ca4f254a0226d0e79485f07d449e2d, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605) passed validation
[2020-01-16 20:30:07.974 INFO     block_validator] Block a15dbc8e646a1b72e799329a31f08df9f00af5684a3cd6a17bf4474209e8fcfd49453eb8c9a9021c7de20ab4b0471e5c06168195b51c18d0e257d44fb41dd67f (block_num:255, state:54e14f1bbe6240c269fd5eb7c8df53069aa545573bc549a861d1eab3715923cd, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605) passed validation
[2020-01-16 20:51:06.213 DEBUG    completer] Drop duplicate block: fc970efb85604cbc0459231b955cf42d3d1e367df9656aeb2405f3eb5adbb41d286788f4b8d986a35a3c23931263e37150eb78c093b5e3cb0bb1bdcb3277060d (block_num:255, state:54e14f1bbe6240c269fd5eb7c8df53069aa545573bc549a861d1eab3715923cd, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605)
[2020-01-16 20:51:06.222 INFO     block_validator] Block fc970efb85604cbc0459231b955cf42d3d1e367df9656aeb2405f3eb5adbb41d286788f4b8d986a35a3c23931263e37150eb78c093b5e3cb0bb1bdcb3277060d (block_num:255, state:54e14f1bbe6240c269fd5eb7c8df53069aa545573bc549a861d1eab3715923cd, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605) passed validation
[2020-01-16 22:01:24.417 ERROR    block_validator] Block validation failed with unexpected error: f60446e169c4e89107e7b87a16656d20b86bf369205b8bd590e421bd6007bfb3544a03e7e275bbe7893d8515598ecd2723e38ab925bf18f55af5b2e29aae9b40 (block_num:255, state:594bc75fee12b893b4ca108b7f337459bb186b73669b4759a3f501658297edfe, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605)
[2020-01-16 22:03:59.883 ERROR    block_validator] Block validation failed with unexpected error: 2165edb44a8cb32d6508ce168ed091046d71128bce7d15fe761ea914e49a42e57457539cb6c41411a396571a915e788b94f2a604e19742fdf559f1907a16a821 (block_num:255, state:ef916de7f166530085c6a6cb48ef7c3cf6b20c9add1e6c76b5fddf0666c94709, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605)
[2020-01-16 22:06:35.250 ERROR    block_validator] Block validation failed with unexpected error: e3f71be31fcde325af8d57e999eb83294d87558231d87382465674661c34c3600b08ef500135a73b87b9d004fda54570561a4f73ba7e2fb0e4cf1a2998fde21b (block_num:255, state:7b42124107d0c13811768b2d3094c9d325c9f79805f45b54b0fe171e5ac3661d, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605)
[2020-01-16 22:09:33.160 DEBUG    completer] Drop duplicate block: ad4636ed6a3a7d1b9f1ef94666ced8635aca1366f5ef0d388808cc3880a40caa28ac71f90d14a711e3c576672168f86b46241d61aca6464c457e65b2450491cf (block_num:255, state:e44ba9d193ce2cb2ac32998f231450fd41d886500e031000987ce9c54cc4734f, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605)
[2020-01-16 22:09:33.175 ERROR    block_validator] Block validation failed with unexpected error: ad4636ed6a3a7d1b9f1ef94666ced8635aca1366f5ef0d388808cc3880a40caa28ac71f90d14a711e3c576672168f86b46241d61aca6464c457e65b2450491cf (block_num:255, state:e44ba9d193ce2cb2ac32998f231450fd41d886500e031000987ce9c54cc4734f, previous_block_id:ee7c54afa8467c3d40fee73698026a9193173b3e08bcdac33376d5dd5787f9e0399ca2a8f7c6b088b349f530021b223b7c09c9d7c81ea1c6fae004cd22d29605)


PBFT log: As you can see, Block 255 came for seq 255 and then it came multiple times after for various other sequences (per the spec the seq number should be the block number. )
INFO  | pbft_engine::node:58 | (PP, view 25393, seq 255): Got BlockNew: 255 / 10b2eac6d898f20a015922565a8c31bc739b4392a9fbda27b58b5d035f5516b3608e13757db2e291babfa51bed2dd5543fb5bd607bd2c1acf2cf09c23b995fa0
INFO  | pbft_engine::node:58 | (PP, view 34240, seq 429): Got BlockNew: 255 / a15dbc8e646a1b72e799329a31f08df9f00af5684a3cd6a17bf4474209e8fcfd49453eb8c9a9021c7de20ab4b0471e5c06168195b51c18d0e257d44fb41dd67f
INFO  | pbft_engine::node:58 | (PP, view 34288, seq 431): Got BlockNew: 255 / fc970efb85604cbc0459231b955cf42d3d1e367df9656aeb2405f3eb5adbb41d286788f4b8d986a35a3c23931263e37150eb78c093b5e3cb0bb1bdcb3277060d
INFO  | pbft_engine::node:58 | (PP, view 34594, seq 446): Got BlockNew: 255 / 33a2be8b3440aa1160604f64a42af17c846e781bae98c014f2dc01c8a8741d7418b01ccb1f62821a76b90e4cd59279526772c149f0b34a4faf7e9ecb0cf11a1f
INFO  | pbft_engine::node:58 | (PP, view 56920, seq 560): Got BlockNew: 255 / e0fa86124fc657b750b9b374934d066ed6c33745a1ff563ea786b568ccde54c108f267f9f3b141a5876a5054b6b97efb8b4492d5e0f1f59a456370b0ee27d68b
INFO  | pbft_engine::node:58 | (PP, view 56926, seq 560): Got BlockNew: 255 / 8f990175676d9fef4c357dfa25c9502fbe5b0c474f02228856b439b812e045d863415a770b3627cb71d87a36a3d81ca770b1e2dec4caeeed8993e66d3962bc94
INFO  | pbft_engine::node:58 | (PP, view 56932, seq 562): Got BlockNew: 255 / dcb689787a343edaba3e9666a3737998bd3892de5034437ce8bf3808deb4f7bf37f4cbe0e8a89a3be4608b845be07b9a68619edbefe413f879b32aa7ebbadf98
INFO  | pbft_engine::node:58 | (PP, view 57046, seq 562): Got BlockNew: 255 / 05a8701bd8a585ae2e44730ab8d2e08de38f9edfcaa0e82ca4393b42e1641c525f527527e81dcbff4ce4afe9465fc7ba4cc4799c95bc7d8a98033b8d0544d433
INFO  | pbft_engine::node:58 | (PP, view 57316, seq 568): Got BlockNew: 255 / d88a7a69ec00895564297f09bb0eba3d6432fe24fa8a7005565f5b0b67bfd74a0b79802176ec96582ca13718f4859ffdc27acdf27a6455a579244a149ad3e7c9
INFO  | pbft_engine::node:58 | (PP, view 57322, seq 569): Got BlockNew: 255 / 5ac39ecc5b2473ffea5c73dd77eff9f3fa8333a106f4458ead1a5c0ef39722503b9802c66f6348a69ed6ad5be342b60e6248bcc4e3bd501546fc153e671076a9
INFO  | pbft_engine::node:58 | (PP, view 57352, seq 572): Got BlockNew: 255 / 603e636c4c8f66a3b86a31a226fb22ea17bee05daad603c1916248e8f042bc2a295ea990b12ed56c4135ab51ddc856375f545f13d14196c311c15d21c4dd23f6